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APPARATUS AND METHOD FOR DETERMINING SPATIAL 

ORIENTATION 



TECHNICAL FIELD 

The present invention relates to methods and apparatuses 
for determining the spatial relationship of an input 
apparatus to a surface. 

BACKGROUND 

5 For the input of information into computerized equipment , 
use is made in many applications of, for example, a 
pointing device, often referred to as a computer mouse, 
or digitizing pens. A feature common to input devices of 
these types is that in most cases they supply only 
10 information which can be directly related to two spatial 
dimensions . 

A typical computer mouse is moved on top of a plane 
surface, and coded information about the movement of the 
mouse on the surface is supplied to a computer via mecha- 
15 nical rolling arrangements, electromechanical circuits 
and coding logic. 

A digitizing pen can be designed with an optical or 
electric sensor in its tip, and contact with a digitiz- 
ing tablet designed for the purpose generates signals, 
20 analog or digital, which can be interpreted in a computer 
in terms of the contact position of the pen tip on the 
tablet. 

In some applications, it is also desirable to obtain 
information from a pointing device or a digitizing pen 
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about, for example, its inclination or, more generally, 
its position in relation to the surface with which it is 
in contact or to which it is near. 

According to the known art, this has often been achieved 
5 by means of relatively complicated electrical 

arrangements in the pointing device and a base intended 
specifically for the pointing device. An example of 
this known art can be seen in US Patent Specification 
5,198,623 which discloses a method in a digitizing 

10 arrangement for establishing the tilting angle of a pen 
forming part of the arrangement. The pen has a coil 
which, in interaction with an array of electrical con- 
ductors in a digitizing tablet, generates electrical 
pulses. The electrical pulses are analyzed with regard to 

15 pulse form, a measurement thus being obtained of the tilt 
of the pen in relation to the surface of the tablet. 

The arrangements and the method disclosed in US 5,198,623 
comprise a great many electrical components, which can be 
seen as a disadvantage when simple and inexpensive 
20 solutions are being sought. 

It can also be seen as a disadvantage that it is 
necessary for the pen to interact with a special digitiz- 
ing tablet. A user is therefore tied to a specific combi- 
nation of pen and base. 

25 A method and an apparatus for recording data from a sheet 
is disclosed in US 5,101,096. Data in the form of optical 
recording dots arranged in perpendicular lines and 
columns are detected by a two dimensional (CCD) optical 
line sensor. An inclination angle is calculated between 

30 the optical recording lines and the optical line sensor. 
The inclination angle is then used to restore the 
detected data. 
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However, the method and arrangement disclosed in US 
5,101,096 is only directed to solve a problem of 
compensating for a misalignment between the optical 
recordings arranged in straight lines and the CCD- 
5 detector elements that are arranged in similar lines and 
columns. US 5,101,096 hence fails to show how to 
calculate any misalignment due to the lines of the 
recordings and the detector not being parallel. 

A generally emphasized problem related to the previously 
10 known art is therefore how a more general determination 
of spatial relationship between, for example, a 
digitizing pen and a patterned surface is to be made 
possible in a flexible and simple manner. 

SUMMARY OF THE INVENTION 

15 One object of the present invention is therefore to solve 
problems related to the previously known art. This is 
achieved by means of apparatuses according to claims 1 
and 15 below, and methods and computer programs according 
to claims 29, 41, 42 and 54 as well as a use according to 

20 claim 55. 

According to a first aspect of the invention, an 
apparatus and a corresponding method for determining a 
three-dimensional spatial relationship between a surface 
provided with a predetermined pattern and the apparatus 

25 are disclosed. A part of the surface is imaged in the 
apparatus, after which the image is compared with the 
predetermined pattern. The comparison produces at least 
one reference measurement, by means of which it is 
possible to determine the spatial relationship expressed 

30 in at least the parameters which define the orientation 
of the surface. By using knowledge of the predetermined 
pattern together with an algebraic model of the image 
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formation by the apparatus, a numerical adaptation can be 
carried out. Parameters obtained from the adaptation can 
then be used to calculate the spatial relationship 
between the apparatus and the surface in terms of, for 
5 example, a distance between the apparatus and the surface 
or an angle between the surface and an axis extending 
through the apparatus. In a preferred embodiment, said 
adaptation will be discussed in detail when a surface 
with a dot pattern is used. 

10 One effect of the invention is therefore that, by 

comparing a predetermined pattern with an image of the 
pattern, an assessment of, for example, the rotation, 
tilt and skew of the apparatus as well as the distance 
between the apparatus and the patterned surface can be 

15 obtained. As is known in the art, the orientation of a 
device such as a pen with respect to the pattern on the 
surface can be expressed using three numbers: rotation, 
tilt and skew. The skew is defined as the angle of 
rotation of the pen around its rotational axis. The tilt 
20 angle is the angle the pen extends with respect to a 
normal vector to the surface and the rotation is the 
angle of rotation of the pen with respect to the normal 
vector. A zero tilt implies that the rotation is equal to 
the skew. 

25 One advantage of an apparatus and a corresponding method 
as above is of course that no complicated electrical 
arrangement is required, as is the case in the known art. 
Only processing means, which are suitably programmed- by 
software, are required in order to obtain the desired 

30 measurement of the spatial positional relationship 
between the apparatus and the surface. 
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According to a second aspect of the present invention, an 
apparatus and a method for position determination are 
disclosed. An image is produced of one partial surface of 
a number of partial surfaces on a surface which is 
5 provided with a position-coding pattern. Image-processing 
is carried out with regard to this image, which comprises 
locating a predetermined plurality of symbols in the 
image and determining the value of each of these 
predetermined symbols. The symbols comprise a raster 
10 point and at least one marking, the raster point forming 
part of a raster which extends over the surface, and the 
value of each symbol being indicated by the position of 
said marking in relation to a raster point. 

The position-coding pattern which has been imaged is 
15 separated into a first position code for a first coordi- 
nate for the partial surface and a second position code 
for a second coordinate for the partial surface. The 
first coordinate is then calculated by means of the first 
position code, and the second coordinate is calculated by 
20 means of the second position code. A third coordinate is 
calculated by comparing the imaged part of the pattern 
with the predetermined pattern, at least one reference 
measurement being obtained, which depends on the 
orientation of the surface. Reference measurements 
25 determine the spatial relationship expressed in at least 
the parameters which define the orientation of the sur- 
face. This spatial relationship then determines the third 
coordinate. 

The first two coordinates are thus obtained by 
30 interpreting the symbols, where displacements of the 
markings forming part of the symbols in relation to a 
normal position contain information in the form of the 
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two-dimensional positions on the surface read. The third 
coordinate is calculated by means of the information 
obtained from the distortion of the known pattern which 
occurs on image formation by the apparatus, the image 
5 formed being for the most part affected by the relative 
spatial positional relationship between the reading appa- 
ratus and the patterned surface. For example, the pat- 
tern in the image formed is distorted in perspective when 
reading is carried out in a direction which does not lie 
10 in the normal direction in relation to the surface. More- 
over, the imaged pattern is changed in terms of scale 
when reading is carried out with the apparatus located at 
a greater distance from the surface. 

In a corresponding manner to that in the first aspect 
15 described above, a reference measurement can be used in 
order to calculate the three-dimensional spatial 
relationship between the apparatus and the surface. With 
this, it is possible to determine, for example, a 
distance between the apparatus and the surface or an 
20 angle between the surface and an axis extending through 
the apparatus. These measurements can then be used in a 
mathematically simple manner for determining the third 
coordinate. 

According to this aspect of the invention, a pointing 
25 device can therefore be produced which can supply 

coordinates which represent three dimensions to, for 
example, a computer. The application in the computer 
using the three-dimensional positional information, 
including rotation, tilt and skew can of course be of 
30 different types. Particularly interesting are appli- 
cations relating to calligraphy. Calligraphy applications 
include those applications of drawing/writing where the 
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device is used as a brush. In these applications the 
rotation, tilt and skew of the device, as determined in 
accordance with the invention, may be used to represent, 
on a display or when printing, a digitized trail made 
5 with the device. If combined with an assumed or selected 
shape of a virtual brush, such a trail will on screen or 
in print, have the characteristics of a trail made 
manually by a person using a brush, i.e. a trail with 
varying widths. It is also possible to utilize, e.g., the 
10 information regarding the distance between the pen and 

the surface, as determined by the invention, to represent 
a color density of the trail when presenting it on a 
screen or on paper, thus further enhancing the likeness 
with a manually painted trail. 

15 Moreover, general applications relating to biometrics, 
i.e. measuring of the movement of, e.g., a hand which 
holds and moves the device. Such an application is 
signature recognition . 

Advantages of the invention include the fact that it can 
20 be implemented in an apparatus without moving parts and 
does not require the use of complicated, expensive bases 
in the form of digitizing tablets full of electronics. An 
apparatus according to the invention is preferably 
produced using optical components and an image-processing 
25 processor which reads a pattern on, for example, a sheet 
of paper, the complexity of which in terms of physical 
construction is, to say the least, reduced in comparison 
with a digitizing tablet according to the known art. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 shows schematically an embodiment of a product 
30 provided with a position-coding pattern, 
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Figs 2a-2d show schematically how the symbols can 
be designed in an embodiment of the invention, 

Fig. 3 shows schematically an example of 4x4 symbols used 
to code a position, 

5 Fig. 4 shows schematically an apparatus according to the 
present invention, which can be used for position 
determination in three dimensions, 

Fig. 5 shows schematically how a pattern on a surface is 
imaged in an apparatus according to the invention, and 

10 Fig. 6 shows how a pattern is distorted in an image 
formed in an apparatus according to the invention. 

DESCRIPTION OF PREFERRED EMBODIMENTS 

For the sake of clarity, the detailed . description below 
of the invention has been divided into a number of part 
descriptions. By way of introduction, a coding pattern 

15 will be described with reference to Figs 1, 2a-d and 3. 
As outlined above, this coding pattern can represent 
positional information but can also represent other 
information. After the description of the coding pattern, 
an apparatus intended to read the pattern will then be 

20 described with reference to Fig.. 4. The way a pattern can 
be used for calculating spatial orientation of an appara- 
tus reading the pattern will then be described with refe- 
rence to Figs 5 and 6. 

Although only one example of a coding pattern will be 
25 used in illustrating the invention, it is possible to 

make use of any other suitable coding pattern. Examples 
of such patterns are to be found in US-A-5, 852, 434 
(Sekendur), US-A-5, 051, 736 (IBM), EP-A-0 469 864 (Xerox) 
as well as our own disclosure WO 00/73983, which is 
30 hereby included by reference. Moreover, the coding 
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pattern used below is described in some more detail in 
our own disclosures PCT/SE00/01895, PCT/SE00/01897 and WO 
01/16691 hereby also included by reference. 

Fig. 1 shows a part of a product in the form of a sheet 
5 of paper 1 which, on its surface 2, is provided with an 
optically readable position-coding pattern 3 enabling 
position determination. The position-coding pattern 
consists of symbols 4 which are arranged systematically 
across the surface 2 so as to make its appearance 

10 "patterned". The sheet of paper has an x-coordinate axis 
and a y-coordinate axis. In this case, position determi- 
nation can be carried out on the entire surface of the 
product. In other cases, the surface allowing position 
determination may constitute a smaller part of the pro- 

15 duct. The sheet of paper can be used, for example, to 

produce an electronic representation of information which 
is written or drawn on the surface. The electronic repre- 
sentation can be produced by continuously determining, 
while writing on the surface with a pen, the position of 

20 the pen on the sheet of paper by reading the position- 
coding pattern. 

The position-coding pattern comprises a virtual raster 
which therefore neither is visible to the human eye nor 
can be detected directly by an apparatus which is to 

25 determine positions on the surface, and a plurality of 

symbols 4 which are each capable of assuming one of four 
values "i"-"4" as. will be described below. It should be 
pointed out here that, for the sake of clarity, the 
position-coding pattern in Fig. 1 has been greatly 

30 enlarged. Furthermore, the position-coding pattern is 
shown on only part of the sheet of paper. 



WO 01/71654 PCT/SE01/00610 

10 

The position-coding pattern is arranged in such a manner 
that the position of a partial surface on the writing 
surface is coded by the symbols on this partial surface. 
A first and a second partial surface 5a, 5b are indicated 
5 by dashed lines in Fig. 1. That part of the position- 
coding pattern (in this case 3x3 symbols) which is 
present on the first partial surface 5a codes a first 
position, and that part of the position-coding pattern 
which is present on the second partial surface 5b codes a 
10 second position. The position-coding pattern is therefore 
partly common to the adjoining first and second 
positions. Such a position-coding pattern is referred to 
in this application as "floating". 

Figs 2a-d show an embodiment of a symbol which can be 
15 used in the position-coding pattern according to the 

invention. The symbol comprises a virtual raster point 
6 which is represented by the intersection between the 
raster lines, and a marking 7 which is in the form of a 
dot. The value of the symbol depends on where the marking 
20 is located. In the example in Fig. 2, there are four pos- 
sible positions, one on each of the raster lines extend- 
ing from the raster points. The displacement from the 
raster point is equal for all the values. In the follow- 
ing, the symbol has the value 1 in Fig. 2a, the value 2 
25 in Fig. 2b, the value 3 in Fig. 2c and the value 4 in 

Fig. 2d. In other words, there are four different types 
of symbol. 

Each symbol can thus represent four values "1-4". This 
means that the position-coding pattern can be divided 
30 into a first position code for the x-coordinate, and a 
second position code for the y-coordinate . The division 
is effected as follows: 



WO 01/71654 



PCT/SE01/00610 



11 



Symbol value 


x-code 


y-code 


1 


1 


1 


2 


0 


1 


3 


1 


0 


4 ! 


0 


0 



10 



15 



20 



25 



The value of each symbol is therefore translated into a 
first digit, in this case bit, for the x-code and a 
second digit, in this case bit, for the y-code. In this 
manner, two completely independent bit patterns are 
obtained. The patterns can be combined to form a common 
pattern which is coded graphically by means of a plura- 
lity of symbols according to Fig. 2. 

Each position is coded by means of a plurality of 
symbols. In this example, use is made of 4x4 symbols 
to code a position in two dimensions, that is to say an 
x-coordinate and a y-coordinate. 

The position code consists of a number series of ones and 
zeros which have the characteristic that no sequence of 
four bits appears more than once in the series. The 
number series is cyclic, which means that 
the characteristic also applies when the end of the 
series is connected to its beginning. Thus a four-bit 
sequence always has an unambiguously determined position 
in the number series. 

The series can be at most 16 bits long if it is to have 
the characteristic described above for sequences of four 
bits. In this example, however, use is made of a series 
having a length of only seven bits as follows: 
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"0001010". 

This series contains seven unique sequences of four bits 
which code a position in the series as follows: 



Position in the series 


Sequence 


0 


0001 


1 . 


0010 


2 


0101 


3 


1010 


4 


0100 


5 


1000 


6 


0000 



5 

For coding the x-coordinate, the number series is written 
sequentially in columns across the entire surface that is 
to be coded. The coding is based on the difference or 
positional displacement between numbers in adjacent 

10 columns. The size of the difference is determined by the 
position (that is to say the sequence) in the number 
series, with which the column is made to begin. More spe- 
cifically, taking the difference modulo seven between, 
on the one hand, a number which is coded by a four-bit 

15 sequence in a first column and which can thus have the 
value (position) 0-6, and, on the other hand, a corre- 
sponding number (that is to say the sequence on the same 
"level") in an adjacent column, the result will be the 
same irrespective of where along the two columns the com- 

20 parison is made. By means of the difference between two 
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columns, it is therefore possible to code an x-coordinate 
which is constant for all y-coordinates . 

Since each position on the surface is coded with 
4x4 symbols in this example, three differences (having 
5 the value 0-6) as stated above are available to code the 
x-coordinate. The coding is then carried out in such a 
manner that, of the three differences, one will always 
have the value 1 or 2 and the other two will have values 
in the range 3-6. Consequently, no differences are allow- 
10 ed to be zero in the x-code. In other words the x-code 

is structured so that the differences will be as follows: 

(3-6) (3-6) (1-2) (3-6) (3-6) (1-2) (3-6) (3-6) (1-2)... 

Each x-coordinate is therefore coded with two numbers 
between 3 and 6 and a subsequent number which is 1 or 2 . 
15 If three is subtracted from the high numbers and one from 
the low, a number in mixed base will be obtained, which 
directly yields a position in the x-direction, from which 
the x-coordinate can then be determined directly, as 
shown in the example below. 

- 

20 By means of the principle described above, it is thus 
possible to code x-coordinates 0,1,2 etc. by means of 
numbers representing three differences. These differences 
are coded with a bit pattern which is based on the number 
series above. The bit pattern can finally be coded 

25 graphically by means of the symbols in Fig. 2. 

In many cases, when reading 4x4 symbols, a complete 
number which codes the x-coordinate will not be produc- 
ed, but parts of two numbers. Since the least significant 
part of the numbers is always 1 or 2, however, a complete 
30 number can easily be reconstructed. 
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The y-coordinates are coded according to the same 
principle as is used for the x-coordinates . The cyclic 
number series is repeatedly written in horizontal rows 
across the surface which is to be position-coded. In 
5 exactly the same way as for the x-coordinates, the rows 
are made to begin in different positions, that is to say 
with different sequences, in the number series. However, 
for the y-coordinates, use is not made of differences, 
but coordinates are coded with numbers that are based on 

10 the starting position of the number series on each row. 
This is because, when the x-coordinate for 4x4 symbols 
has been determined, it is possible to determine the 
starting positions in the number series for the rows that 
are included in the y-code in the 4x4 symbols. In the 

15 y-code, the most significant digit is determined by let- 
ting this be the only one that has a value in a specific 
range. In this example, one row of four is made to begin 
in the position 0-1 in the number series to indicate that 
this row relates to the least significant digit in a 

20 y-coordinate, and the other three are made to begin in 
the position 2-6. In the y-direction, there is thus a 
series of numbers as follows: 

(2-6) (2-6) (2-6) (0-1) (2-6) (2-6) (2-6) (0-1) (2-6)... 

Each y-coordinate is thus coded with three numbers 
25 between 2 and 6 and a subsequent number between 0 and 1. 

If 0 is subtracted from the low number and 2 from the 
high numbers, a position in the y-direction in mixed 
base, from which it is possible to determine the y-coor- 
dinate directly, is obtained in the same manner as for 
30 the x-direction. 

With the above method, it is possible to code 

4 x 4 x 2 = 32 positions in the x-direction. Each such 
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position corresponds to three differences, which gives 
3 x 32 = 96 positions. Moreover, it is possible to code 
5x5x5x2= 250 positions in the y-direction, 
Each such position corresponds to 4 rows, which gives 
5 4 x 250 = 1000 positions. Altogether it is thus possible 
to code 96000 positions. Since the x-coding is based on 
differences, however, it is possible to select the posi- 
tion in which the first number series begins. Taking 
account of the fact that this first number series can 

10 begin in seven different positions, it is possible to 

code 7 x 96000 = 672000 positions. The starting position 
of the first number series in the first column can be 
calculated when the x-coordinate has been determined. 
The above-mentioned seven different starting positions 

15 for the first series may code different sheets of paper 
or writing surfaces on a product. 

With a view to further illustrating the invention 
according to this embodiment, a specific example follows, 
which is based on the described embodiment of the posi J 
20 tion code. 

Fig. 3 shows an example of an image with 4x4 symbols 
which are read by an apparatus for position determina- 
tion. 

These 4x4 symbols have the following values: 
25 4 4 4 2 

3 2 3 4 

4 4 2 4 
13 2 4 

These values represent the following binary x- and 
30 y-codes: 
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x-code: 



y-code : 



0 0 0 0 



0 0 0 1 



10 10 



0 10 0 



0 0 0 0 



0 0 10 



5 



110 0 



10 10 



The vertical x-sequences code the following positions in 
the number series: 2 0 4 6. The differences between the 
columns will be -2 4 2, which modulo 7 gives: 5 4 2, 
which in mixed base codes the position (5-3) x 8 + (4-3) 

10 x 2 + (2-1) =16+2+1-19. Since the first coded 

x-position is position 0, the difference which is in the 
range 1-2 and which can be seen in the 4x4 symbols is the 
twentieth such difference. Since, furthermore, there is a 
total of three columns for each such difference and there 

15 is a starting column, the vertical sequence furthest to 

the right in the 4x4 x-code belongs to the 61st column in 
the x-code (3 x 20 + 1 = 61) and the one furthest to the 
left belongs to the 58th. 

The horizontal y-sequences code the positions 0 4 1 3 in 
20 the number series. Since these series begin in the 58th 
column, the starting position of the rows is these 
numbers minus 57 modulo 7, which yields the starting 
positions 6 3 0 2. Translated into digits in the mixed 
base, this will be 6-2, 3-2, 0-0, 2-2 =4100, where 
25 the third digit is the least significant digit, in the 
number in question. The fourth digit is then the most 
significant digit in the next number. In this case, it 
must be the same as in the number in question. (The 
exception is when the number in question consists of 
30 the highest possible digits in all positions. It is 
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then known that the beginning of the next number is one 
greater than the beginning of the number in question.) 

The position of the four-digit number will then be 
0x50 + 4x10 + 1x2 + 0x1 = 42 in the mixed base. 

5 The third row in the y-code is thus the 43rd which has 
the starting position 0 or 1, and, since there are four 
rows in all on each such row, the third row is number 
43x4=172. 

In this example, the position of the uppermost left 
10 corner for the 4x4 symbol group is thus (58,170). 

Since the x-sequences in the 4x4 group begin on row 170, 
the x-columns of the entire pattern begin in the 
positions of the number series ((2046) -169) modulo 
7=1635. Between the last starting position (5) and 

15 the first starting position, the numbers 0-19 are coded 
in the mixed base, and, by adding up the representations 
of the numbers 0-19 in the mixed base, the total diffe- 
rence between these columns is obtained. A primitive 
algorithm for carrying this out is to generate these 

20 twenty numbers and directly add up their digits. The 

resulting sum is called s. The sheet of paper or writing 
surface will then be identified by (5-s) modulo 7. 

In the example above, an embodiment has been described, 
in which each position is coded with 4x4 symbols, and a 

25 number series with 7 bits is used. Of course, this is 

only an example. Positions can be coded with a greater or 
smaller number of symbols. The number of symbols need not 
be the same in both directions. The number series can be 
of different length and need not be binary, but may be 

30 based on another base. Different number series can be 
used for coding in the x-direction and coding in the 
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y-direction. The symbols can have different numbers of 
values . 

In the example above, the marking is a dot but may, of 
course, have a different appearance. For example, it may 
5 consist of a dash which begins in the virtual raster 
point and extends from it to a defined position. 

In the example above, the symbols within a square partial 
surface are used for coding a position. The partial 
surface may have a different shape, for example 
10 hexagonal. The symbols need not be arranged in rows and 
columns at an angle of 90° to each other but can also be 
arranged in some other manner. 

For the position code to be detectable, the virtual 
raster must be determined. This can be carried out by 
15 studying the distance between different markings. The 

shortest distance between two markings must derive from 
two neighboring symbols having the value 1 and 3 so that 
the markings are located on the same raster line between 
two raster points. When such a pair of markings has been 
detected, the associated raster points can be determined 
with knowledge of the distance between the raster points 
and the displacement of the markings from the raster 
points. Once two raster points have been located, addi- 
tional raster points can be determined by means of mea- 
25 sured distances to other markings and with knowledge of 
the distance of the raster points from one another. 

An embodiment of an apparatus for position determination, 
the spatial relationship of which to a surface can be 
determined, is shown schematically in Fig. 4. The 
apparatus comprises a casing 11 having approximately the 
shape of a pen. In one short side of the casing there is 
an opening 12. The short side is intended to bear against 
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or be held a short distance from a surface S on which 
position determination is to be carried out. In the 
Figure, a normal direction v, to the surface S and an 
axis A extending through the apparatus are indicated. 
5 The axis A forms an angle of inclination, or tilt, 9 to 
the normal direction v, . 

The casing contains essentially an optics part, an 
electronics part, and a power supply. 

The optics part comprises at least one light emitting 
10 diode 13 for illuminating the surface which is to 

be imaged and a light-sensitive area sensor 14, such as 
a CCD or CMOS sensor, for recording a two-dimensional 
image. The apparatus may also comprise a lens system 
including a schematically illustrated lens 21. 

15 The power supply to the apparatus is obtained from 

a battery 15 which is mounted in a separate compartment 
in the casing. 

The electronics part comprises image-processing means 16 
for determining a position on the basis of the image 
20 recorded by the sensor 14 and, more specifically, 

a processor unit with a processor which is programmed to 
read images from the sensor and to carry out position 
determination on the basis of these images. 

In this embodiment, the apparatus also comprises a pen 
25 point 17, by means of which it is possible to write 

ordinary pigment-based writing on the surface on which 
position determination is to be carried out. The pen 
point 17 is extendable and retractable so that the user 
can control whether or not it is to be used. In certain 
30 applications, the apparatus need not have a pen point at 
all or comprise a fixed, non-retractable, pen point. 
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Moreover, the device may comprise a plurality of 
retractable pen points, each having, e.g., a different 
color. 

Moreover, the apparatus comprises buttons 18 by means of 
5 which the apparatus is activated and controlled. It also 
comprises a transceiver 19 for wireless transmission, for 
example using IR light or radio waves, of information to 
and from the apparatus. The apparatus can also comprise a 
display 20 for showing positions or recorded information. 
10 Applicant's International Patent application WO 98/20446 
describes a device for recording text. This device can be 
used for position determination if programmed in a 
suitable way. If it is to be used for pigment-based 
writing, it must also have a pen point. 

15 The device can be divided into various physical casings, 
a first casing containing components required for 
capturing images of the position-coding pattern and for 
transferring them to components which are located in a 
second casing and carry out position determination on the 

20 basis of the recorded image or images. 

As mentioned above, position determination is carried out 
by a processor which must therefore have software to 
locate and decode the symbols in an image and 
to determine positions on the basis of the codes thus 
25 obtained. A person skilled in the art can, starting from 
the example above, design software which carries out 
position determination on the basis of an image of part 
of a position-coding pattern. 

Moreover, the skilled person can, on the basis of the 
above description, design software for printing the 
position-coding pattern. 
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In the embodiment above, the pattern is optically 
readable and the sensor is therefore optical. As mention- 
ed above, the pattern can be based on a parameter other 
than an optical parameter. Obviously, in that case the 
5 sensor must be of a type which can read the parameter in 
question. Examples of such parameters are of chemical, 
acoustical or electromagnetic character, e.g. resistance, 
capacitance and inductance. 

In the embodiment above, the raster is a rectangular 
10 grid. It can also have other forms, i.e. non-rectangular. 

In the embodiment above, the longest possible cyc- 
lic number series is not used. Thus, there is a certain 
amount of redundancy which can be used, for example, to 
check the rotation of the group of symbols read. 

15 With reference to Figs 5 and 6, an explanation follows of 
how a grid of markings, or dots, is distorted on image 
formation and how this distortion is used for calculating 
the spatial orientation of an apparatus reading the 
pattern or, more correctly, the relative spatial 

20 orientation between the patterned surface and the reading 
apparatus . 

The actual calculations are performed in processing 
means, such as those discussed above in connections with 
figure 4, that is a processor within the device itself. 

25 It is also feasible to perform the calculations in an 

external processor connected to the reading device. The 
software which performs the calculations is written in 
accordance with the mathematical expressions to follow 
and it is assumed that the person skilled in the art will 

30 choose suitable tools for performing such programming. 

The software may be present, or stored, in any form known 
in the art, such as any volatile or non-volatile memory 
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units capable of being connected to the processor, such 
as a diskette or a CD-ROM, as well as propagated signals 
such as the stream of bits that represent Internet 
transmission of packets or the carrier waves that are 
5 transmitted to satellites. 

Fig. 6a shows schematically the pattern 601 as it appears 
when it is applied to a surface. The pattern is 
preferably in the form of such a position-coding pattern 
as is described above in connection with Figs 1-3. While 

10 it is true that, in the patterns used as examples pre- 
viously, a plurality of the markings are displaced in 
relation to an orthogonal raster, these displacements are 
assumed to be relatively small and thus of minor signi- 
ficance in this embodiment. The displacements may also, 

15 however, be part of a larger plurality of predetermined 
vectors which in that case are also incorporated in the 
calculations . 

In the discussion to follow, it is assumed that the 
sensor (i.e. sensor 14 in figure 4) is arranged such that 
20 its normal direction is parallel with the direction of 
extension of the device. The person skilled in the art 
may adapt the calculations to allow for other relations 
between the sensor normal direction and the device. 

The points in the square grid, as illustrated in Fig. 6a, 
25 are imaged through a lens 503, e.g. lens 21 in figure 4, 
on the surface 502 of a sensor located in the reading 
apparatus. The pattern 602 on the sensor surface is, as 
shown in Fig. 6b, distorted because the relative spatial 
orientation between the reading apparatus and the 
30 patterned surface is not orthogonal. 
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In this case, the pattern is characterized in that 
inherent stretches are described by the predetermined 
two-dimensional vectors 

{a:a = k xi x + k yi y\ 

where k xif k yi are integers and x,y are two-dimensional base 
vectors. This pattern is to be compared with the distort- 
ed image on the sensor surface. 

10 In order to model the image formation through the lens 

system, a coordinate system is selected which is fixed in 
relation to the sensor and the lens and in which the lens 
is located at the origin. It is assumed that all the 
light rays from the surface to the sensor pass 

15 unrefracted through the origin. The degrees of freedom of 
the model are the distance to and the orientation of the 
patterned surface. 

A point P 0 on the sensor is selected. The stretch from P 0 
to the lens is defined by the vector v 0 . In the same 
20 manner, a vector v, belonging to the point P± is defined. 
Given this model of image formation, a vector v k from the 
origin to the point P k in the image on the sensor will be 
parallel to the vector v\ from the origin to the 
corresponding point P' k on the patterned surface: 

25 

Two spatial vectors v x and v y can be introduced, which 
lie in the plane of the patterned surface and constitute 
base vectors for the grid as it is rotated in space. By 
30 using the predetermined vectors, the vector from the 
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point P' 0 in the pattern on the surface to any other 
point P'i is, expressed in v, and v yt given by: 



=kj x +k yl v y 

5 where the integers k xit k yi describe the integer position 
of a point in the grid relative to P' 0 . We can thus 
describe a relationship according to Fig. 5: 



10 By introducing the vectors 
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the following is obtained: 
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An equation for each dimension is then: 
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Division results in identical right-hand parts: 
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For each point P if there is therefore an equation system 
of the type: 

+ **v„ + c« V = ^ + + c„v w = a rf + ^ + c- (1) 

where the unknowns are: 

( v ~ v xv v^)sv, and (v^ v yz )^v y 

If the number of points is greater than five, there is a 
redundant equation system in, for example, v^v^v^ andv„„ 

xx xy yx yy 
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The least square method gives v^v^v^ and . v^and v xy 
are inserted into equation (1) and a similar equation 
system is obtained for and v >T which is also solved by 
means of the least square method. 
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The direction of the vectors v x and v is thus known, and 

in this way the spatial orientation of the pattern read 
has been determined. 

The vectors v x and v y obtained then give a normal vector 
5 to the patterned surface through the cross product 

v, = v x x . The angle 9 between the normal vector v, and 
the z axis, which is fixed in the apparatus, is given by 



COS0 = 




Usually, as is known in the art, the orientation of a 
10 device such as a pen with respect to the pattern on the 
surface can be expressed using three numbers: rotation, 
tilt and skew. The skew is defined as the angle of 
rotation of the pen around its rotational axis. The tilt 
angle is the angle the pen extends with respect to a 
15 normal vector to the surface and the rotation is the 

angle of rotation of the pen with respect to the normal 
vector. A zero tilt implies that the rotation is equal to 
the skew. These angles are illustrated in figure 4, where 
the normal vector to the surface is denoted N, rotation 
20 is denoted R, the tilt is denoted 6 and the skew is 

denoted SK. Note that in figure 4, the axis A of the 
device illustrates a coinciding optical axis and axis of 
rotation of the device. 

In order to calculate rotation, tilt and skew, we begin 
25 by noting from above that 

1 * . _ 1 - 
v x = — v x and v =— v y 

from which equations, c 0 can be calculated. 
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A vector v'o extending from the origin to the surface on 
which the pattern is located, parallel with the vector v 0 
extending from P 0 to the origin, is given by 

v'o = c 0 v 0 

5 We also know from above that the normal vector to the 
surface, extending away from the pen, is given by 

* * A 

Now an equation for the plane of the surface can be 
calculated. The distance from the origin to the surface 
10 is given by 



For each vector v p = (x p9 y p9 z p ) from the origin to the plane 
of the surface the expression 




15 holds, which gives an equation for the plane of the 
surface : 

The axis of the pen is assumed to be parallel with the z- 
axis. Hence, a vector extending from the origin to the 
20 surface, parallel with the axis of the pen can be 

calculated by inserting x p =0 and y p =0 in the equation for 
the plane of the surface: 

v 0p =(0,0,-/> 0 /P 2 ) 

The angle between the normal vector and v Qp gives the 
25 tilt: 
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If the tilt is non-zero, the skew can be calculated. The 
projection of the normal vector onto the xy-plane gives: 

5 Then the skew is given by the angular deviation from the 
y-axis : 



'V(Q,i,o)^ 



± skew = cos 1 



where the skew is positive when is positive and 
negative otherwise. 

10 The projection of the axis of the pen onto the plane of 
the surface is now obtained by: 

P*={proj x9 proj y ) 

where 

P r °J x = v 0p • v x and proj y = v 0p ■ v y 

15 The angle between this vector and the vectors of the 
pattern gives the rotation: 



rot = ± cos 



P r °j x 

l w 



and, if the tilt is zero, we have: 



rot = ±cos" 



20 where a positive y-coordinate implies a positive 
rotation. 



WO 01/71654 



PCT/SE01/00610 



29 

The position of the tip of the pen can be calculated by 
considering the fact that, when the pen changes 
orientation, during writing for example, the equation for 
the plane of the surface will change. By considering such 
5 a changing equation for the plane of the surface as a 

sequence of different planes in which the tip of the pen 
is located, the position of the tip can simply be found 
by noting that only one point will be common to all 
planes: the point of the tip of the pen. This calculation 
10 can be performed when there are available at least three 
different equations for the plane of the surface. 

A plane is defined by an equation, as indicated above, by 
it's coefficients. For a plane i, we have: 

{fxi y Pyx 1 > P Z i 9 ) 

15 The position of the tip of the pen is given by a system 
of equations: 

P*X* +P yi y +P :X Z +^0, =o 

P«* +P yk y +P :k z +/ > 0 *=o 

which can be solved in accordance with any procedure 
known in the art, yielding the xyz-coordinates, a 
20 distance vector, for the tip of the pen. 

The exact position of the physical tip of the pen may 
then be utilized to find the exact displacement between 
the tip and the position calculated from the position 
coding pattern, i.e. the center of the images obtained. 
25 Such a displacement is inherent and, in essence, 

unavoidable since the optical axis of the lens system of 
the pen does not coincide with the writing pen. The 
displacement is not necessarily known in advance due to 
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mechanical inaccuracies when assembling the pen as well 
as inaccuracies occurring when performing refill- 
replacement-actions in the case the pen is equipped with 
a replaceable refill defining the writing tip. In a pen 
5 equipped with multiple tips, for example of different 
colors, the location of the tip with respect to the 
sensor will vary for each color. Such variations in tip 
location can be determined , and accounted for, by way of 
the present invention. 

10 The calculated displacement is preferably used to 

synchronize the physical track made by the pen and the 
digitally recorded track as determined by the center of 
the images. This is advantageous in that it obviates the 
need to mechanically measure and calibrate the spatial 

15 relation between the tip of the pen and the optical 
system. 
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CLAIMS 

1. An apparatus for determining a three-dimensional 
spatial relationship between a surface (S) provided with 
a predetermined pattern (601) and the apparatus, 
5 comprising: 

- means for imaging a part of the pattern, 

- means for comparing the imaged part (602) of the 
pattern with the predetermined pattern, at least one 
reference measurement being obtained, which depends on 
the orientation ( v x , v y9 v z ) of the surface, 

- means for determining, by means of the reference 
measurement, the spatial relationship expressed in at 
least the parameters which define the orientation of the 
surface . 

15 2. An apparatus according to claim 1, wherein the means 
for comparison comprise means for comparing between a 
number of directed stretches in the image and predeter- 
mined vectors which follow from the predetermined 
pattern. 

20 3. An apparatus according to claim 2, comprising means 
for adaptation according to the least square method. 

4 . An apparatus according to any one of the preceding 
claims, wherein the spatial relationship comprises a 
distance vector between the apparatus and the surface 

25 from which the pattern has been imaged. 

5. An apparatus according to any one of the preceding 
claims, wherein the spatial relationship at least 
comprises an angle between an axis extending through the 
apparatus and the surface from which the pattern has been 

30 imaged. 
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6. An apparatus according to any one of claims 1-5, 
wherein the means for comparison comprise means for 



determining a set of parameters which define a vector 
which, in relation to a plane extending through the appa- 
5 ratus, unambiguously determines an inclination for the 
surface from which the pattern has been imaged. 

7. An apparatus according to any one of claims 1-5, 
wherein the means for comparison comprise means for 
determining a set of parameters which define a vector 

10 which unambiguously determines a normal vector for the 
surface from which the pattern has been imaged. 

8. An apparatus according to any one of claims 1-5, 
wherein the means for comparison comprise means for 
determining at least one parameter which unambiguously 

15 defines an angular orientation for the imaged pattern 
about a normal vector for the surface from which the 
pattern has been imaged. 

9. An apparatus according to any one of claims 1-8, 
wherein the means for comparison comprise means for 

20 determining at least one parameter which unambiguously 
defines at least one of rotation, tilt and skew of the 
apparatus . 

10. An apparatus according to any one of claims 1-5, 
wherein the imaging means comprise means for one-dimen- 

25 sional pattern imaging. 

11. An apparatus according to any one of claims 1-9, 
wherein the imaging means comprise means for two-dimen- 
sional pattern imaging. 

12. An apparatus according to any one of claims 1-11, 
30 wherein the apparatus is hand-held. 
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13. An apparatus according to any one of claims 1-12, 
wherein the apparatus has means (19) for wireless 
transmission of information. 

14. An apparatus according to any one of claims 1-13, 
5 wherein the apparatus is in the general form of a pen and 

comprises means for determining the position of a tip of 
the pen. 

15. An apparatus for position determination, comprising a 
sensor (14) for producing an image of one partial surface 
of a plurality of partial surfaces on a surface, which is 
provided with a position-coding pattern, and image- 
processing means (16), which are arranged to 

- locate a predetermined plurality of symbols in the 
image, each symbol comprising a raster point (5) and at 

15 least one marking (6), the raster point forming part of 
a raster which extends over the surface and the value of 
each symbol being indicated by the position of said mark- 
ing in relation to a raster point, 

- determine the value of each of said predetermined 
20 plurality of symbols, 

- separate the position-coding pattern in the image 
into a first position code for a first coordinate for 
the partial surface and a second position code for a 
second coordinate for the partial surface by translating 

25 the value of each symbol into at least one first digit 

for the first position code and at least one second digit 
for the second position code, 

- calculate the first coordinate by means of the 
first position code and the second coordinate by means 
of the second position code, 

- compare the imaged part of the pattern (602) with 
the predetermined pattern (601), at least one reference 
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measurement being obtained, which depends on the orienta- 
tion (v x ,v y ,v z ) of the surface, 

- determine, by means of the reference measurement, 
a three-dimensional spatial relationship between the 

5 surface and the apparatus expressed in at least the para- 
meters which define the orientation of the surface, 

- calculate a third coordinate, use being made of 
the measurement of the spatial relationship. 

16. An apparatus according to claim 15, wherein the 

10 image-processing means are also arranged to carry out a 
comparison between a number of directed stretches in the 
image and predetermined vectors which follow from the 
predetermined pattern. 

17. An apparatus according to claim 16, wherein the 
15 image-processing means are also arranged to carry out 

adaptation according to the least square method. 

18. An apparatus according to any one of claims 15-17, 
wherein the spatial relationship comprises a distance 
vector between the apparatus and the partial surface from 

20 which the position-coding pattern has been imaged. 

19. An apparatus according to any one of claims 15-18, 
wherein the spatial relationship at least comprises an 
angle between an axis extending through the apparatus and 
the partial surface from which the position-coding 

25 pattern has been imaged. 

20. An apparatus according to any one of claims 15-19, 
wherein the means for comparison comprise means for 
determining a set of parameters which define a vector 
which, in relation to a plane extending through the appa- 

30 ratus, unambiguously determines an inclination for the 
surface from which the pattern has been imaged. 
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21. An apparatus according to any one of claims 15-19, 
wherein the image-processing means are also arranged to 
determine a set of parameters which define a vector which 
unambiguously determines a normal vector for the surface 

5 from which the pattern has been imaged. 

22. An apparatus according to any one of claims 15-19, 
wherein the image-processing means are also arranged to 
determine at least one parameter which unambiguously 
defines an angular orientation for the imaged pattern 
about a normal vector for the partial surface from which 
the pattern has been imaged. 

23. An apparatus according to any one of claims 15-22, 
wherein the means for comparison comprise means for 
determining at least one parameter which unambiguously 

15 defines at least one of rotation, tilt and skew of the 
apparatus . 

24. An apparatus according to any one of claims 15-19, 
wherein the image-processing means are also arranged for 
one-dimensional pattern imaging. 

25. An apparatus according to any one of claims 15-22, 
wherein the image-processing means are also arranged 
for two-dimensional pattern imaging. 

26. An apparatus according to any one of claims 15-25, 
wherein the apparatus is hand-held. 

25 27. An apparatus according to any one of claims 15-26, 
wherein the apparatus has means (19) for wire- 
less transmission of information. 

28. An apparatus according to any one of claims 15-27, 
wherein the apparatus is in the general form of a pen and 
30 comprises means for determining the position of a tip of 
the pen. 
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29. A method for determining a three-dimensional spatial 
relationship between a surface (S) provided with a 
predetermined pattern (601) and an apparatus capable of 

5 at least imaging the surface, comprising: 

- imaging a part of the pattern, 

- comparing the imaged part (602) of the pattern 
with the predetermined pattern, at least one reference 
measurement being obtained, which depends on the orien- 

10 tation (v x9 v y9 v g ) of the surface, 

- determining, by means of the reference measure- 
ment, the spatial relationship expressed in at least the 
parameters which define the orientation of the surface. 

30. A method according to claim 29, wherein the 

15 comparison is carried out between a number of directed 
stretches in the image and predetermined vectors which 
follow from the predetermined pattern. 

31. A method according to claim 30, wherein the 
adaptation is carried out according to the least square 

20 method. 

32. A method according to any one of claims 29-31, 
wherein the spatial relationship comprises a distance 
vector between the apparatus and the surface from which 
the pattern has been imaged. 

25 33. A method according to any one of claims 29-32, 

wherein the spatial relationship at least comprises an 
angle between an axis extending through the apparatus and 
the surface from which the pattern has been imaged. 

34. A method according to any one of claims 29-33, 
30 wherein the comparison comprises determining a set of 
parameters which define a vector which, in relation to 
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a plane extending through the apparatus, unambiguously 
determines an inclination for the surface from which the 
pattern has been imaged. 

35. A method according to any one of claims 29-33, 



parameters which define a vector which unambiguously 
determines a normal vector for the surface from which 
the pattern has been imaged. 

36. A method according to any one of claims 29-33, 

10 wherein the comparison comprises determining at least one 
parameter which unambiguously defines an angular orienta- 
tion for the imaged pattern about a normal vector for the 
surface from which the pattern has been imaged. 

37. A method according to any one of claims 29-36, 

15 wherein the comparison comprise determining at least one 
parameter which unambiguously defines at least one of 
rotation, tilt and skew of the apparatus. 

38. A method according to any one of claims 29-33, 
wherein the image formation comprises one-dimensional 

20 pattern imaging. 

39. A method according to any one of claims 29-36, 
wherein the image formation comprises two-dimensional 
pattern imaging. 

40. A method according to any one of claims 29-37, 

25 wherein the apparatus is in the general form of a pen and 
the method comprises determining the position of a tip of 
the pen. 

41. A computer program which is stored in a computer- 
readable memory medium and which comprises instructions 

30 for making a computer carry out the method according to 
any one of claims 29-40. 



5 wherein the comparison comprises determining a set of 



• 



WO 01/71654 PCT/SE01/00610 

38 

42. A method for position determination, comprising 
producing an image of one partial surface of a plurality 
of partial surfaces on a surface, which is provided with 
a position-coding pattern, and carrying out image- 
5 processing, which comprises 

- locating a predetermined plurality of symbols in 
the image, each symbol comprising a raster point (5) and 
at least one marking (6), the raster point forming part 
of a raster which extends over the surface and the value 

10 of each symbol being indicated by the position of said 
marking in relation to a raster point, 

- determining the value of each of said predetermin- 
ed plurality of symbols, 

- separating the position-coding pattern in the 

15 image into a first position code for a first coordinate 
for the partial surface and a second position code for a 
second coordinate for the partial surface by translating 
the value of each symbol into at least one first digit 
for the first position code and at least one second digit 

20 for the second position code, 

- calculating the first coordinate by means of the 
first position code and the second coordinate by means of 
the second position code, 

- comparing the imaged part of the pattern (602) 

25 with the predetermined pattern (601), at least one refe- 
rence measurement being obtained, which depends on the 
orientation (v x9 v y9 v.) of the surface, 

- determining, by means of the reference measure- 
ment, a three-dimensional spatial relationship between 

30 the surface and the apparatus expressed in at least the 
parameters which define the orientation of the surface, 

- calculating a third coordinate, use being made of 
the spatial relationship. 
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43. A method according to claim 42, wherein the com- 
parison is carried out between a number of directed 
stretches in the image and predetermined vectors which 
follow from the predetermined pattern. 

5 44. A method according to claim 43, wherein the 

adaptation is carried out according to the least square 
method. 

45. A method according to any one of claims 42-44, 
wherein the spatial relationship comprises a distance 

10 vector between the apparatus and the partial surface from 
which the position-coding pattern has been imaged. 

46. A method according to any one of claims 42-45, 
wherein the spatial relationship at least comprises an 
angle between an axis extending through the apparatus and 

15 the partial surface from which the position-coding 
pattern has been imaged. 

47. A method according to any one of claims 42-47, 
comprising determining a set of parameters which define 
a vector which, in relation to a plane extending through 

20 the apparatus, unambiguously determines an inclination 
for the surface from which the pattern has been imaged. 

48. A method according to any one of claims 42-46, 
comprising determining a set of parameters which define a 
vector which unambiguously determines a normal vector for 

25 the surface from which the pattern has been imaged. 

49. A method according to any one of claims 42-46, 
comprising determining at least one parameter which unam- 
biguously defines an angular orientation for the imaged 
pattern about a normal vector for the partial surface 

30 from which the pattern has been imaged. 
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50. A method according to any one of claims 42-4 9, 
wherein the comparison comprise determining at least one 
parameter which unambiguously defines at least one of 
rotation, tilt and skew of the apparatus. 

5 51. A method according to any one of claims 42-46, 

wherein the image formation comprises one-dimensional 
pattern imaging. 

52. A method according to any one of claims 42-49, 
wherein the image formation comprises two-dimensional 

10 pattern imaging. 

53. A method according to any one of claims 42-52, 
wherein the apparatus is in the general form of a pen and 
the method comprises determining the position of a tip of 
the pen. 

15 54. A computer program which is stored in a computer- 
readable memory medium and which comprises instructions 
for making a computer carry out the method according to 
any one of claims 42-53. 



20 



55. Use of distortion in an image of a patterned surface 
for calculation of the relative spatial orientation 
between the surface and an apparatus capturing said 
image . 

56. Use according to claim 55, wherein said relative 
spatial orientation includes a rotation and/or a tilt 

25 and/or a skewing and/or a distance between the surface 
and the apparatus. 

57. Use according to claim 55 or 56, wherein said 
apparatus is a digital pen. 
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